@sphuber this is awesome 👍.

It would also be perfect if it was possible to use this for the test suite. Perhaps there could be some pytest switch that would allow changing the storage backend? It is also currently a bit annoying that the backend is loaded unconditionally, even if you run just a single test file that does not require it. Again, a pytest switch that disables the session fixture that loads the backend would be awesome (and I suspect would speed up iteration during development).

Thanks for the review @giovannipizzi

I fully support Daniel's comment - maybe open an issue and we look into it once this is merged?

Regarding switching off the automatic loading of the backend in the session fixture: we had discussed this in the past, but decided against it since it is used by plugins and so changing it would be breaking, as they would have to start adding it manually. I agree that having it off by default would still be better, so maybe we can revisit whether this is an acceptable breaking change for plugins.

As for changing the backend plugin: I already have the required scaffolding in place in the aiida-s3 plugin. We can repurpose this and add it to aiida-core.

I also opened aiidateam/aiida-tutorials#468 for the future, when we'll prepare the next tutorial
Finally, one important note. In the previous coding week, we've been discussing having a documentation page for the various backends, and have it properly linked in the docs so people see it when they setup AiiDA. And this was combined with a simpler verdi profile setup mechanism. I think @superstar54 , @unkcpz and/or @eimrek might have been working on this? Has this ever been merged? I recommend adding the documentation for these new backend as well. Either we open an issue now and plan to fix it soon, or if that documentation is already in the docs, we could fix it directly in this PR

I was not aware of this effort but agree that this would be very useful. This ties neatly into why we cannot merge this PR just yet. There is still some functionality that is not supported by this backend, mostly related to the querybuilder:

• use of date_trunc in aiida.storage.psql_dos.orm.querybuilder.QueryBuilder.get_creation_statistics
• queries that use JSONB syntax
• Group.count()
  If I cannot fix these, then we should clearly document this at the bare minimum, and we should stress that this backend should really only be used for testing, demoing and other non-production type activities.

Since this PR was anyway dependent on a bunch of other PRs being merged first, I had put these final details on hold. But will try to find time soon to work on them.

Update:
The following tests of tests/orm/test_querybuilder.py fail with the sqlite backend:

FAILED tests/orm/test_querybuilder.py::TestRepresentations::test_as_sql - AssertionError: FILES DIFFER:
FAILED tests/orm/test_querybuilder.py::TestRepresentations::test_as_sql_inline - AssertionError: FILES DIFFER:
FAILED tests/orm/test_querybuilder.py::TestRepresentations::test_as_sql_literal_quote - ValueError: SQLite does not support JSON query: aliased(DbNode).extras.['elements'] contains ['Si']
FAILED tests/orm/test_querybuilder.py::test_analyze_query - ValueError: SQLite does not support JSON query: aliased(DbNode).extras.['key'] contains ['x', 1]
FAILED tests/orm/test_querybuilder.py::TestAttributes::test_attribute_existence - AssertionError: assert {'38c68cf9-3c...2c8b5eb7bf06'} == set()
FAILED tests/orm/test_querybuilder.py::TestAttributes::test_attribute_type - AssertionError: assert {'c940c2c7-00...32e7360d1638'} == {'df904228-8d...32e7360d1638'}
FAILED tests/orm/test_querybuilder.py::TestManager::test_statistics - sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such function: date_trunc
FAILED tests/orm/test_querybuilder.py::TestManager::test_statistics_default_class - sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such function: date_trunc

It seems the part of JSON query functionality that is not supported includes at least the contains operator. There also seem to be problems with the JSONB queries respecting type. The !has_key notation also doesn't seem to work as expected.

I think @superstar54 , @unkcpz and/or @eimrek might have been working on this? Has this ever been merged?

@giovannipizzi I missed this comment, sorry.
I think you mentioned the #6070 draft done by @superstar54 during coding week if I remember correctly. Not sure there are overlap work but it was mentioned the Sqlite backend is in aiida-core already.

Not sure there are overlap work but it was mentioned the Sqlite backend is in aiida-core already.

The PR of @superstar54 uses the SqliteTempBackend which already exists in aiida-core. Here I am adding another backend based on sqlite though, the SqliteDosStorage. The difference is that the latter is persistent, whereas the SqliteTempBackend gets destroyed as soon as the Python interpreter shuts down.

This is blocked by #6173 Currently it is necessary to specify --first-name, --last-name and --institution with verdi profile setup -n making it way less easy to create a new profile. This should be fixed first.